In the field of a digital still camera or a digital video camera, there is known an image synthesis technique in which a plurality of image data obtained through continuous shooting by changing an exposure condition such as a shutter speed or a f-number are synthesized to create an image having a wide dynamic range. In such an image synthesis technique using the continuous shooting, a method of performing position alignment or correcting a positional deviation between images becomes important in order to alleviate an influence of a blur caused by handshaking or an object movement generated during the continuous shooting.
While various position alignment methods are known in the art, a template matching method is typically employed in an embedded system. In the template matching method, an area (template) having a predetermined size (for example, 8 pixels by 8 lines) is defined in one (reference image) of a pair of images as a position alignment target, and a matching index value representing a matching degree between the template and the comparison target image is sequentially computed by shifting the template within a predetermined range (search range) of the other image (comparison target image). In addition, a relative positional deviation between images is obtained by searching an area having the highest matching index value (or lowest value depending on a type of the matching index value). The matching index value may include a sum of squared intensity difference (SSD), a sum of absolute intensity difference (SAD) of the error, a normalized cross-correlation (NCC), and the like.
The SSD is defined in the following Equation (1). Determination is made such that the matching degree increases as the SSD decreases.
                                          R            SSD                    =                                    ∑                              j                =                0                                            N                -                1                                      ⁢                                          ∑                                  i                  =                  0                                                  M                  -                  1                                            ⁢                                                (                                                            I                      ⁡                                              (                                                  i                          ,                          j                                                )                                                              -                                          T                      ⁡                                              (                                                  i                          ,                          j                                                )                                                                              )                                2                                                    ,                            (        1        )                            where a size of the template is set to M×N [pixels], and        T(i, j) and I(i, j) denote a luminance value of the template in a template position (i, j) and a luminance value of the comparison target image, respectively.        
The SAD is defined in the following Equation (2). Determination is made such that the matching degree increases as the SAD decreases.
                              R          SAD                =                              ∑                          j              =              0                                      N              -              1                                ⁢                                    ∑                              i                =                0                                            M                -                1                                      ⁢                                                                          I                  ⁡                                      (                                          i                      ,                      j                                        )                                                  -                                  T                  ⁡                                      (                                          i                      ,                      j                                        )                                                                                                                        (        2        )            
The NCC is defined in the following Equation (3). Determination is made such that the matching degree increases as the NCC increases.
                              R          NCC                =                                            ∑                              j                =                0                                            N                -                1                                      ⁢                                          ∑                                  i                  =                  0                                                  M                  -                  1                                            ⁢                                                I                  ⁡                                      (                                          i                      ,                      j                                        )                                                  ·                                  T                  ⁡                                      (                                          i                      ,                      j                                        )                                                                                                                          ∑                                  j                  =                  0                                                  N                  -                  1                                            ⁢                                                ∑                                      i                    =                    0                                                        M                    -                    1                                                  ⁢                                                                            I                      ⁡                                              (                                                  i                          ,                          j                                                )                                                              2                                    ×                                                            ∑                                              j                        =                        0                                                                    N                        -                        1                                                              ⁢                                                                  ∑                                                  i                          =                          0                                                                          M                          -                          1                                                                    ⁢                                                                        T                          ⁡                                                      (                                                          i                              ,                              j                                                        )                                                                          2                                                                                                                                                    (        3        )            
The normalized cross-correlation method is advantageous in that images having different brightness levels can be directly used as a processing image because a normalization term is included in the index value. However, a computation amount disadvantageously increases. Meanwhile, in the SAD or SSD method, basically, it is assumed that images having matching brightness levels are input. However, since a computation amount is relatively small, the SAD or SSD method is widely employed.
A technique of obtaining an image having a wide dynamic range by performing a position alignment process and a synthesis process for images obtained through continuous shooting under different exposure conditions is also disclosed in JP5-7336A, JP2008-277896A, JP2007-202098A, and JP2007-272459A.
In the technique disclosed in JP5-7336A, an image having a wide dynamic range is obtained by synthesizing a plurality of images obtained by performing shooting under different exposure conditions, and a synthesis process is performed after a time-dependent positional deviation is corrected.
In the technique of JP2008-277896A, shooting is alternately performed to obtain a short-exposure image and a long-exposure image, and it is determined that a long-exposure image having a smaller absolute difference out of the long-exposure images obtained before and after the short-exposure image is suitable for the synthesis, so that the long-exposure image having the smaller absolute difference is selected as a synthesis image.
In the technique disclosed in JP2007-202098A, a synthesized image is obtained by performing a synthesis process after correcting the positional deviation between the short-exposure image and the long-exposure image. In the synthesis process, a blur is suppressed by changing a synthesis ratio from a pair of differential images while an edge blur and a noise are separated using a threshold value.
In the technique disclosed in JP2007-272459A, in an imaging device capable of obtaining a wide dynamic range by synthesizing a plurality of images obtained under different exposure conditions, reliability of the displacement is obtained based on a relative displacement between images, a plurality of images used in the synthesis are selected based on the obtained reliability, position alignment is performed for a plurality of the selected images based on the displacement of the images, and the synthesis is performed.